{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 关键词搜索（进阶版）\n",
    "\n",
    "Kaggle竞赛题：https://www.kaggle.com/c/home-depot-product-search-relevance\n",
    "\n",
    "鉴于课件里已经完整的show了NLTK在各个NLP处理上的用法，我这里就不再重复使用了。\n",
    "\n",
    "本篇的教程里会尽量用点不一样的库，让大家感受一下Python NLP领域各个库的优缺点。\n",
    "\n",
    "同时，\n",
    "\n",
    "在进阶版故事中，我们将讨论除了一些显而易见的“自制feature”之外，更加牛x的算法：\n",
    "\n",
    "- String Distance\n",
    "\n",
    "- TF-iDF\n",
    "\n",
    "- Word2Vec（具体原理会在下堂课继续）\n",
    "\n",
    "注意，前面的预处理部分维持不变（懒得折腾。。预处理这事儿 要玩儿可以玩儿一年。。）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step1：导入所需"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "所有要用到的库"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.ensemble import RandomForestRegressor, BaggingRegressor\n",
    "from nltk.stem.snowball import SnowballStemmer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "读入训练/测试集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df_train = pd.read_csv('../input/train.csv', encoding=\"ISO-8859-1\")\n",
    "df_test = pd.read_csv('../input/test.csv', encoding=\"ISO-8859-1\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里还有个有用的玩意儿，叫产品介绍"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df_desc = pd.read_csv('../input/product_descriptions.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "看看数据们都长什么样子"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>product_uid</th>\n",
       "      <th>product_title</th>\n",
       "      <th>search_term</th>\n",
       "      <th>relevance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>100001</td>\n",
       "      <td>Simpson Strong-Tie 12-Gauge Angle</td>\n",
       "      <td>angle bracket</td>\n",
       "      <td>3.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "      <td>100001</td>\n",
       "      <td>Simpson Strong-Tie 12-Gauge Angle</td>\n",
       "      <td>l bracket</td>\n",
       "      <td>2.50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9</td>\n",
       "      <td>100002</td>\n",
       "      <td>BEHR Premium Textured DeckOver 1-gal. #SC-141 ...</td>\n",
       "      <td>deck over</td>\n",
       "      <td>3.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>16</td>\n",
       "      <td>100005</td>\n",
       "      <td>Delta Vero 1-Handle Shower Only Faucet Trim Ki...</td>\n",
       "      <td>rain shower head</td>\n",
       "      <td>2.33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>17</td>\n",
       "      <td>100005</td>\n",
       "      <td>Delta Vero 1-Handle Shower Only Faucet Trim Ki...</td>\n",
       "      <td>shower only faucet</td>\n",
       "      <td>2.67</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id  product_uid                                      product_title  \\\n",
       "0   2       100001                  Simpson Strong-Tie 12-Gauge Angle   \n",
       "1   3       100001                  Simpson Strong-Tie 12-Gauge Angle   \n",
       "2   9       100002  BEHR Premium Textured DeckOver 1-gal. #SC-141 ...   \n",
       "3  16       100005  Delta Vero 1-Handle Shower Only Faucet Trim Ki...   \n",
       "4  17       100005  Delta Vero 1-Handle Shower Only Faucet Trim Ki...   \n",
       "\n",
       "          search_term  relevance  \n",
       "0       angle bracket       3.00  \n",
       "1           l bracket       2.50  \n",
       "2           deck over       3.00  \n",
       "3    rain shower head       2.33  \n",
       "4  shower only faucet       2.67  "
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>product_uid</th>\n",
       "      <th>product_description</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>100001</td>\n",
       "      <td>Not only do angles make joints stronger, they ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>100002</td>\n",
       "      <td>BEHR Premium Textured DECKOVER is an innovativ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>100003</td>\n",
       "      <td>Classic architecture meets contemporary design...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>100004</td>\n",
       "      <td>The Grape Solar 265-Watt Polycrystalline PV So...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>100005</td>\n",
       "      <td>Update your bathroom with the Delta Vero Singl...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   product_uid                                product_description\n",
       "0       100001  Not only do angles make joints stronger, they ...\n",
       "1       100002  BEHR Premium Textured DECKOVER is an innovativ...\n",
       "2       100003  Classic architecture meets contemporary design...\n",
       "3       100004  The Grape Solar 265-Watt Polycrystalline PV So...\n",
       "4       100005  Update your bathroom with the Delta Vero Singl..."
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_desc.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "看来不要做太多的复杂处理，我们于是直接合并测试/训练集，以便于统一做进一步的文本预处理"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df_all = pd.concat((df_train, df_test), axis=0, ignore_index=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>product_title</th>\n",
       "      <th>product_uid</th>\n",
       "      <th>relevance</th>\n",
       "      <th>search_term</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>Simpson Strong-Tie 12-Gauge Angle</td>\n",
       "      <td>100001</td>\n",
       "      <td>3.00</td>\n",
       "      <td>angle bracket</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "      <td>Simpson Strong-Tie 12-Gauge Angle</td>\n",
       "      <td>100001</td>\n",
       "      <td>2.50</td>\n",
       "      <td>l bracket</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9</td>\n",
       "      <td>BEHR Premium Textured DeckOver 1-gal. #SC-141 ...</td>\n",
       "      <td>100002</td>\n",
       "      <td>3.00</td>\n",
       "      <td>deck over</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>16</td>\n",
       "      <td>Delta Vero 1-Handle Shower Only Faucet Trim Ki...</td>\n",
       "      <td>100005</td>\n",
       "      <td>2.33</td>\n",
       "      <td>rain shower head</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>17</td>\n",
       "      <td>Delta Vero 1-Handle Shower Only Faucet Trim Ki...</td>\n",
       "      <td>100005</td>\n",
       "      <td>2.67</td>\n",
       "      <td>shower only faucet</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id                                      product_title  product_uid  \\\n",
       "0   2                  Simpson Strong-Tie 12-Gauge Angle       100001   \n",
       "1   3                  Simpson Strong-Tie 12-Gauge Angle       100001   \n",
       "2   9  BEHR Premium Textured DeckOver 1-gal. #SC-141 ...       100002   \n",
       "3  16  Delta Vero 1-Handle Shower Only Faucet Trim Ki...       100005   \n",
       "4  17  Delta Vero 1-Handle Shower Only Faucet Trim Ki...       100005   \n",
       "\n",
       "   relevance         search_term  \n",
       "0       3.00       angle bracket  \n",
       "1       2.50           l bracket  \n",
       "2       3.00           deck over  \n",
       "3       2.33    rain shower head  \n",
       "4       2.67  shower only faucet  "
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_all.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "合并之后我们得到:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(240760, 5)"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_all.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "产品介绍也是一个极有用的信息，我们把它拿过来："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df_all = pd.merge(df_all, df_desc, how='left', on='product_uid')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>product_title</th>\n",
       "      <th>product_uid</th>\n",
       "      <th>relevance</th>\n",
       "      <th>search_term</th>\n",
       "      <th>product_description</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>Simpson Strong-Tie 12-Gauge Angle</td>\n",
       "      <td>100001</td>\n",
       "      <td>3.00</td>\n",
       "      <td>angle bracket</td>\n",
       "      <td>Not only do angles make joints stronger, they ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "      <td>Simpson Strong-Tie 12-Gauge Angle</td>\n",
       "      <td>100001</td>\n",
       "      <td>2.50</td>\n",
       "      <td>l bracket</td>\n",
       "      <td>Not only do angles make joints stronger, they ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9</td>\n",
       "      <td>BEHR Premium Textured DeckOver 1-gal. #SC-141 ...</td>\n",
       "      <td>100002</td>\n",
       "      <td>3.00</td>\n",
       "      <td>deck over</td>\n",
       "      <td>BEHR Premium Textured DECKOVER is an innovativ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>16</td>\n",
       "      <td>Delta Vero 1-Handle Shower Only Faucet Trim Ki...</td>\n",
       "      <td>100005</td>\n",
       "      <td>2.33</td>\n",
       "      <td>rain shower head</td>\n",
       "      <td>Update your bathroom with the Delta Vero Singl...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>17</td>\n",
       "      <td>Delta Vero 1-Handle Shower Only Faucet Trim Ki...</td>\n",
       "      <td>100005</td>\n",
       "      <td>2.67</td>\n",
       "      <td>shower only faucet</td>\n",
       "      <td>Update your bathroom with the Delta Vero Singl...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id                                      product_title  product_uid  \\\n",
       "0   2                  Simpson Strong-Tie 12-Gauge Angle       100001   \n",
       "1   3                  Simpson Strong-Tie 12-Gauge Angle       100001   \n",
       "2   9  BEHR Premium Textured DeckOver 1-gal. #SC-141 ...       100002   \n",
       "3  16  Delta Vero 1-Handle Shower Only Faucet Trim Ki...       100005   \n",
       "4  17  Delta Vero 1-Handle Shower Only Faucet Trim Ki...       100005   \n",
       "\n",
       "   relevance         search_term  \\\n",
       "0       3.00       angle bracket   \n",
       "1       2.50           l bracket   \n",
       "2       3.00           deck over   \n",
       "3       2.33    rain shower head   \n",
       "4       2.67  shower only faucet   \n",
       "\n",
       "                                 product_description  \n",
       "0  Not only do angles make joints stronger, they ...  \n",
       "1  Not only do angles make joints stronger, they ...  \n",
       "2  BEHR Premium Textured DECKOVER is an innovativ...  \n",
       "3  Update your bathroom with the Delta Vero Singl...  \n",
       "4  Update your bathroom with the Delta Vero Singl...  "
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_all.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "好了，现在我们得到一个全体的数据大表格"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: 文本预处理\n",
    "\n",
    "我们这里遇到的文本预处理比较简单，因为最主要的就是看关键词是否会被包含。\n",
    "\n",
    "所以我们统一化我们的文本内容，以达到任何term在我们的数据集中只有一种表达式的效果。\n",
    "\n",
    "我们这里用简单的Stem做个例子：\n",
    "\n",
    "（有兴趣的同学可以选用各种你觉得靠谱的预处理方式：去掉停止词，纠正拼写，去掉数字，去掉各种emoji，等等）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "stemmer = SnowballStemmer('english')\n",
    "\n",
    "def str_stemmer(s):\n",
    "    return \" \".join([stemmer.stem(word) for word in s.lower().split()])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为了计算『关键词』的有效性，我们可以naive地直接看『出现了多少次』"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def str_common_word(str1, str2):\n",
    "    return sum(int(str2.find(word)>=0) for word in str1.split())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接下来，把每一个column都跑一遍，以清洁所有的文本内容"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df_all['search_term'] = df_all['search_term'].map(lambda x:str_stemmer(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df_all['product_title'] = df_all['product_title'].map(lambda x:str_stemmer(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df_all['product_description'] = df_all['product_description'].map(lambda x:str_stemmer(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: 进阶版文本特征\n",
    "\n",
    "这里我们讨论几个有点逼格的文本特征算法："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Levenshtein\n",
    "\n",
    "使用Levenshtein很直观，直接调用python标准库即可"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.625"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import Levenshtein\n",
    "\n",
    "Levenshtein.ratio('hello', 'hello world')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "好的，接下来我们把search_term和product_title进行比较："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df_all['dist_in_title'] = df_all.apply(lambda x:Levenshtein.ratio(x['search_term'],x['product_title']), axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "同理，对产品介绍进行比较："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df_all['dist_in_desc'] = df_all.apply(lambda x:Levenshtein.ratio(x['search_term'],x['product_description']), axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### TF-iDF\n",
    "\n",
    "TF-iDF稍微复杂点儿，因为它涉及到一个需要提前把所有文本计算统计一下的过程。\n",
    "\n",
    "我们首先搞一个新的column，叫all_texts, 里面是所有的texts。（我并没有算上search term, 因为他们不是一个结构完整的句子，可能会影响tfidf的学习）。为了防止句子格式不完整，我们也强制给他们加上句号。\n",
    "\n",
    "*注意：这里我们最严谨的做法是把train/test先分开，然后只在train上做tfidf的学习，并在test上直接转化。然而由于我不想把整个文章顺序打乱（因为本来train/test分开的步骤在最后），因为我希望大家看的时候可以跟简单版教程有很好的结构对照，所以我直接tfidf用在全部的语料集上。如果想安慰自己的话，你可以这么考虑：我们的算法虽然看到了test的文本内容，但是没有test的label。可以姑且认为test的文本内容本身也就是可见的。并且，这种行为在kaggle的竞赛中并没有被禁止，很多竞赛的kernel提交算法都有这问题，所以我们姑且先不care。但是我个人非常不赞同这么做。同学们如果自己在线下做实验的时候，希望可以做到严谨。不要cheating哦~*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df_all['all_texts']=df_all['product_title'] + ' . ' + df_all['product_description'] + ' . '"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "搞完之后，长这样："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    simpson strong-ti 12-gaug angl . not onli do a...\n",
       "1    simpson strong-ti 12-gaug angl . not onli do a...\n",
       "2    behr premium textur deckov 1-gal. #sc-141 tugb...\n",
       "3    delta vero 1-handl shower onli faucet trim kit...\n",
       "4    delta vero 1-handl shower onli faucet trim kit...\n",
       "Name: all_texts, dtype: object"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_all['all_texts'][:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "然后，我们取出所有的单字，做成一个我们的单词字典：（这里我们用gensim，为了更加细致的分解TFIDF的步骤动作；其实sklearn本身也有简单好用的tfidf模型，详情见第二课stock news基本版教程）\n",
    "\n",
    "Tokenize可以用各家或者各种方法，就是把长长的string变成list of tokens。包括NLTK，SKLEARN都有自家的解决方案。或者你自己直接用str自带的split()方法，也是一种tokenize。只要记住，你这里用什么，那么之后你的文本处理都得用什么。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dictionary(221877 unique tokens: ['fontain', 'extrusion', 'shiftingbreak', 'unfold', 'fibersdesign']...)\n"
     ]
    }
   ],
   "source": [
    "from gensim.utils import tokenize\n",
    "from gensim.corpora.dictionary import Dictionary\n",
    "dictionary = Dictionary(list(tokenize(x, errors='ignore')) for x in df_all['all_texts'].values)\n",
    "print(dictionary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "楼上可见，我们得到了221877个单词的大词典。我们以此准备出一个语料库。\n",
    "因为语料库一般都很大，所以我建议我们做语料的时候都用个iterator来实现。\n",
    "\n",
    "于是我们写一个类(class)：\n",
    "\n",
    "这个类所做的事情也很简单，就是扫便我们所有的预料，并且转化成简单的单词的个数计算。（Bag-of-Words）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class MyCorpus(object):\n",
    "    def __iter__(self):\n",
    "        for x in df_all['all_texts'].values:\n",
    "            yield dictionary.doc2bow(list(tokenize(x, errors='ignore')))\n",
    "\n",
    "# 这里这么折腾一下，仅仅是为了内存friendly。面对大量corpus数据时，你直接存成一个list，会使得整个运行变得很慢。\n",
    "# 所以我们搞成这样，一次只输出一组。但本质上依旧长得跟 [['sentence', '1'], ['sentence', '2'], ...]一样\n",
    "\n",
    "corpus = MyCorpus()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "有了我们标准形式的语料库，我们于是就可以init我们的TFIDFmodel了。\n",
    "\n",
    "这里做的事情，就是把已经变成BoW向量的数组，做一次TFIDF的计算。详情参见课件。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from gensim.models.tfidfmodel import TfidfModel\n",
    "tfidf = TfidfModel(corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "好，这下我们看看一个普通的句子放过来长什么样子："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(985, 0.2947139124944075),\n",
       " (3430, 0.28760732706613895),\n",
       " (33767, 0.6587176730120703),\n",
       " (35249, 0.6296957697663794)]"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfidf[dictionary.doc2bow(list(tokenize('hello world, good morning', errors='ignore')))]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "怎么判断两个句子的相似度呢？\n",
    "\n",
    "这里有个trick，因为我们得到的tfidf只是『有这个字，就有这个值』，并不是一个全部值。\n",
    "\n",
    "也就是说，两个matrix可能size是完全不一样的。\n",
    "\n",
    "想用cosine计算的同学就会问了，两个matrix的size都不fix，怎么办？\n",
    "\n",
    "咦，这里就注意咯。他们的size其实是一样的。只是把全部是0的那部分给省略了对吧？\n",
    "\n",
    "于是，我们只要拿其中一个作为index。扩展开全部的matrixsize，另一个带入，就可以计算了。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from gensim.similarities import MatrixSimilarity\n",
    "\n",
    "# 先把刚刚那句话包装成一个方法\n",
    "def to_tfidf(text):\n",
    "    res = tfidf[dictionary.doc2bow(list(tokenize(text, errors='ignore')))]\n",
    "    return res\n",
    "\n",
    "# 然后，我们创造一个cosine similarity的比较方法\n",
    "def cos_sim(text1, text2):\n",
    "    tfidf1 = to_tfidf(text1)\n",
    "    tfidf2 = to_tfidf(text2)\n",
    "    index = MatrixSimilarity([tfidf1],num_features=len(dictionary))\n",
    "    sim = index[tfidf2]\n",
    "    # 本来sim输出是一个array，我们不需要一个array来表示，\n",
    "    # 所以我们直接cast成一个float\n",
    "    return float(sim[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "来，我们可以做个测试："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8566456437110901"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text1 = 'hello world'\n",
    "text2 = 'hello from the other side'\n",
    "cos_sim(text1, text2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "好，万事俱备，我们现在就只需要把我们的column都转化成tfidf计算出来的相似度了：\n",
    "\n",
    "因为sim的结果是一个np array，所以我们用'[0]'取里面那个唯一值就行了"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df_all['tfidf_cos_sim_in_title'] = df_all.apply(lambda x: cos_sim(x['search_term'], x['product_title']), axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们来看看长什么样子："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    0.274539\n",
       "1    0.000000\n",
       "2    0.000000\n",
       "3    0.133577\n",
       "4    0.397320\n",
       "Name: tfidf_cos_sim_in_title, dtype: float64"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_all['tfidf_cos_sim_in_title'][:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "同理，"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df_all['tfidf_cos_sim_in_desc'] = df_all.apply(lambda x: cos_sim(x['search_term'], x['product_description']), axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "至此，我们又有了两个高质量的features。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Word2Vec\n",
    "\n",
    "最后，我们的大杀器w2v登场了。\n",
    "\n",
    "最好的情况呢，其实是你在另一个地方用非常大非常完备的语料库做好w2v的training，再跑到这个任务上来直接输出vector。具体可以参照官网上谷歌新闻语料的一个大大的w2v模型。可下载，以后用。\n",
    "\n",
    "我们这里，用个轻量级的解决方案，那就是直接在我们的文本上学习。（跟tfidf玩法差不多）\n",
    "\n",
    "然而，w2v和tfidf有个很大的不同。对于tfidf而言，只需要知道一整段text中包含了哪些word元素就行了。其他都不用care。\n",
    "\n",
    "但是w2v要考虑到句子层级的split，以及语境前后的考虑。\n",
    "\n",
    "所以，刚刚tfidf的corpus不能直接被这里使用。在这里，我们需要把句子/文字都给分类好。\n",
    "\n",
    "这里，我们再玩儿一个NLP的杀器：NLTK。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import nltk\n",
    "# nltk也是自带一个强大的句子分割器。\n",
    "tokenizer = nltk.data.load('tokenizers/punkt/english.pickle')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们看看效果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['simpson strong-ti 12-gaug angl .',\n",
       " 'not onli do angl make joint stronger, they also provid more consistent, straight corners.',\n",
       " 'simpson strong-ti offer a wide varieti of angl in various size and thick to handl light-duti job or project where a structur connect is needed.',\n",
       " 'some can be bent (skewed) to match the project.',\n",
       " 'for outdoor project or those where moistur is present, use our zmax zinc-coat connectors, which provid extra resist against corros (look for a \"z\" at the end of the model number).versatil connector for various 90 connect and home repair projectsstrong than angl nail or screw fasten alonehelp ensur joint are consist straight and strongdimensions: 3 in.',\n",
       " 'x 3 in.',\n",
       " 'x 1-1/2 in.mad from 12-gaug steelgalvan for extra corros resistanceinstal with 10d common nail or #9 x 1-1/2 in.',\n",
       " 'strong-driv sd screw . ']"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.tokenize(df_all['all_texts'].values[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "依照这个方法，我们先把长文本搞成list of 句子，再把句子变成list of 单词："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sentences = [tokenizer.tokenize(x) for x in df_all['all_texts'].values]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "其实这些sentences不需要这些层级关系，他们都是平级的，所以：\n",
    "\n",
    "我们把list of lists 给 flatten了。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sentences = [y for x in sentences for y in x]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这下我们可以看到，一共1998321个句子。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1998321"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(sentences)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接着，我们把句子里的单词给分好。这事儿我们还是可以用刚刚Gensim的tokenizer\n",
    "\n",
    "这里我们用nltk的word_tokenizer来试试"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from nltk.tokenize import word_tokenize\n",
    "w2v_corpus = [word_tokenize(x) for x in sentences]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "好，这下我们训练model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from gensim.models.word2vec import Word2Vec\n",
    "\n",
    "model = Word2Vec(w2v_corpus, size=128, window=5, min_count=5, workers=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这时候，每个单词都可以像查找字典一样，读出他们的w2v坐标了："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 4.74717617,  2.64686418,  0.35502195,  0.80881947, -2.7076602 ,\n",
       "       -1.64652073, -2.27783561, -0.92420739, -0.14867701,  5.74324369,\n",
       "       -0.7946381 ,  4.2346034 ,  0.85286433,  2.52050495, -0.73137748,\n",
       "        1.9509604 , -1.75785506, -0.81137753,  1.03638244, -1.19456983,\n",
       "       -0.23727451, -2.1338408 , -0.28541893, -2.99906659, -3.9443922 ,\n",
       "        3.45797253, -1.05518758,  2.46135116, -2.20839262, -1.80382502,\n",
       "        4.74548674,  1.45839429,  1.89756656, -0.03712106,  1.1392957 ,\n",
       "       -0.28602245, -0.8830694 , -0.23989263,  1.52211082, -3.42239022,\n",
       "        0.01539903,  0.87455851,  0.25244436, -1.04864109, -2.5990386 ,\n",
       "       -0.72978145,  3.70196033,  1.23104656,  1.96894944,  0.28224495,\n",
       "       -2.56046462, -1.574512  , -2.20811892,  2.57060266, -2.39487839,\n",
       "       -2.57700634,  2.41890597,  0.75820315, -2.01431084, -2.13502717,\n",
       "       -0.1664609 , -0.10223585, -0.93037611,  4.34696054, -0.88387358,\n",
       "       -4.04143572, -0.76093572, -4.64342451,  5.51188993,  1.15895557,\n",
       "        0.04778517, -2.19732451,  2.58349681,  0.30812854, -0.05851545,\n",
       "        0.14742707,  2.65333915, -3.49614239, -0.04863766, -2.16823196,\n",
       "       -0.75142419,  4.91761446, -0.76077127, -1.32876885,  2.45301771,\n",
       "       -1.47275043,  0.42842156,  0.99622703, -0.10033795,  1.29662645,\n",
       "       -2.37165427, -1.56022704, -2.82586884, -0.85700899,  2.06937575,\n",
       "        1.59914327,  1.34667575,  3.50663733, -0.44751605,  2.495821  ,\n",
       "       -0.80405504,  1.46751213, -2.97747278,  1.59027314,  1.04117584,\n",
       "       -3.76480174,  2.03447819,  2.54762697,  0.32881331, -3.20821214,\n",
       "        2.40063143, -0.48981774, -0.83241153,  1.22963691,  1.81204796,\n",
       "       -0.20612268, -0.78395975, -0.48303336, -1.06597555, -0.19221658,\n",
       "       -0.07359049,  2.20414066,  1.43124104, -0.65005296, -1.06764543,\n",
       "       -0.60594636,  1.24271679, -3.92345881], dtype=float32)"
      ]
     },
     "execution_count": 105,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model['right']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "跟TFiDF一样，我们可以把textual的column都转化为w2v坐标\n",
    "\n",
    "这里不一样的是，TFiDF是针对每个句子都可以有的，而w2v是针对每个单词的。\n",
    "\n",
    "所以我们要有个综合考虑的idea：\n",
    "\n",
    "平均化一个句子的w2v向量，算作整个text的平均vector：\n",
    "\n",
    "我们写一个方法来实现："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# 先拿到全部的vocabulary\n",
    "vocab = model.vocab\n",
    "\n",
    "# 得到任意text的vector\n",
    "def get_vector(text):\n",
    "    # 建立一个全是0的array\n",
    "    res =np.zeros([128])\n",
    "    count = 0\n",
    "    for word in word_tokenize(text):\n",
    "        if word in vocab:\n",
    "            res += model[word]\n",
    "            count += 1\n",
    "    return res/count            "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们可以试试："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.463416    0.97560888  0.10863534 -0.82696981  1.41277812  0.79170082\n",
      "  0.0978372  -0.68150822  0.99790255 -0.44159976  1.00841724 -0.43905148\n",
      "  0.21946578  1.01400004 -1.97788476 -0.24557593 -1.13167921 -0.22579467\n",
      " -0.48376473  0.41615273  0.67180802  0.53008292  0.52145608 -1.9788011\n",
      "  1.42492926 -0.18045649 -0.65986957 -0.93241019  1.1279408   0.06740171\n",
      "  0.40830285  0.79715398 -0.84070924  0.35409186  0.29476608 -0.01593437\n",
      " -0.2836557  -0.27918024  1.09351952 -0.34930461  2.4379538   0.60693575\n",
      " -0.42956293  1.8092562  -0.43648594 -0.58754889 -1.25327353 -0.65632194\n",
      "  1.29220566  0.52857347  0.08685846  0.04752858 -0.59915119 -0.00281862\n",
      " -0.99020389 -0.82765476  0.16989418  1.01381083 -0.0428756   0.98613351\n",
      " -1.3658159  -1.43513804 -0.75343464 -0.09422477  0.07967508 -0.87333807\n",
      "  0.36081878 -0.81233797  0.2529      1.28962335  1.31695796 -1.45198588\n",
      "  1.3620293   0.11232918 -0.21050634 -0.27940779 -0.13681145 -1.03338094\n",
      "  0.07509582  0.46833945  0.00504241  1.29592314  0.38266217  1.22406197\n",
      "  0.49212121 -0.04775342 -0.78706613  0.15082838 -1.38598437  0.85031726\n",
      " -1.25327267 -0.74126085  0.19504826 -1.05701404 -0.18221375 -1.85480102\n",
      "  0.10107571  1.40432313 -1.16039764 -0.56544506  0.47745609  0.47314491\n",
      " -0.52448719 -0.36235778  1.88896974 -0.16886154 -0.01455577 -1.06299005\n",
      "  0.18107549 -0.74452799 -2.65968527  0.5563812  -0.17996723  0.65088516\n",
      " -0.25565534 -1.15591651  0.46034975 -1.45567893 -0.11601611 -0.49641909\n",
      "  1.75690769 -0.28630929 -1.34841273  1.24394214  0.26161586 -0.05742724\n",
      " -0.83873362 -0.8819646 ]\n"
     ]
    }
   ],
   "source": [
    "print(get_vector('life is like a box of chocolate'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "好，同理，我们需要计算两个text的平均w2v的cosine similarity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from scipy import spatial\n",
    "# 这里，我们再玩儿个新的方法，用scipy的spatial\n",
    "\n",
    "def w2v_cos_sim(text1, text2):\n",
    "    try:\n",
    "        w2v1 = get_vector(text1)\n",
    "        w2v2 = get_vector(text2)\n",
    "        sim = 1 - spatial.distance.cosine(w2v1, w2v2)\n",
    "        return float(sim)\n",
    "    except:\n",
    "        return float(0)\n",
    "# 这里加个try exception，以防我们得到的vector是个[0,0,0,...]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.2397940355621142"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w2v_cos_sim('hello world', 'hello from the other side')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "跟刚刚TFIDF一样，我们计算一下search term在product title和product description中的cosine similarity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df_all['w2v_cos_sim_in_title'] = df_all.apply(lambda x: w2v_cos_sim(x['search_term'], x['product_title']), axis=1)\n",
    "df_all['w2v_cos_sim_in_desc'] = df_all.apply(lambda x: w2v_cos_sim(x['search_term'], x['product_description']), axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>product_title</th>\n",
       "      <th>product_uid</th>\n",
       "      <th>relevance</th>\n",
       "      <th>search_term</th>\n",
       "      <th>product_description</th>\n",
       "      <th>dist_in_title</th>\n",
       "      <th>dist_in_desc</th>\n",
       "      <th>all_texts</th>\n",
       "      <th>tfidf_cos_sim_in_title</th>\n",
       "      <th>tfidf_cos_sim_in_desc</th>\n",
       "      <th>w2v_cos_sim_in_title</th>\n",
       "      <th>w2v_cos_sim_in_desc</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2</td>\n",
       "      <td>simpson strong-ti 12-gaug angl</td>\n",
       "      <td>100001</td>\n",
       "      <td>3.00</td>\n",
       "      <td>angl bracket</td>\n",
       "      <td>not onli do angl make joint stronger, they als...</td>\n",
       "      <td>0.190476</td>\n",
       "      <td>0.030418</td>\n",
       "      <td>simpson strong-ti 12-gaug angl . not onli do a...</td>\n",
       "      <td>0.274539</td>\n",
       "      <td>0.182836</td>\n",
       "      <td>0.482381</td>\n",
       "      <td>0.425223</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>3</td>\n",
       "      <td>simpson strong-ti 12-gaug angl</td>\n",
       "      <td>100001</td>\n",
       "      <td>2.50</td>\n",
       "      <td>l bracket</td>\n",
       "      <td>not onli do angl make joint stronger, they als...</td>\n",
       "      <td>0.153846</td>\n",
       "      <td>0.022901</td>\n",
       "      <td>simpson strong-ti 12-gaug angl . not onli do a...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.322772</td>\n",
       "      <td>0.110944</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>9</td>\n",
       "      <td>behr premium textur deckov 1-gal. #sc-141 tugb...</td>\n",
       "      <td>100002</td>\n",
       "      <td>3.00</td>\n",
       "      <td>deck over</td>\n",
       "      <td>behr premium textur deckov is an innov solid c...</td>\n",
       "      <td>0.175000</td>\n",
       "      <td>0.017875</td>\n",
       "      <td>behr premium textur deckov 1-gal. #sc-141 tugb...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.053455</td>\n",
       "      <td>0.307271</td>\n",
       "      <td>0.445303</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>16</td>\n",
       "      <td>delta vero 1-handl shower onli faucet trim kit...</td>\n",
       "      <td>100005</td>\n",
       "      <td>2.33</td>\n",
       "      <td>rain shower head</td>\n",
       "      <td>updat your bathroom with the delta vero single...</td>\n",
       "      <td>0.326087</td>\n",
       "      <td>0.048632</td>\n",
       "      <td>delta vero 1-handl shower onli faucet trim kit...</td>\n",
       "      <td>0.133577</td>\n",
       "      <td>0.043712</td>\n",
       "      <td>0.532707</td>\n",
       "      <td>0.465209</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>17</td>\n",
       "      <td>delta vero 1-handl shower onli faucet trim kit...</td>\n",
       "      <td>100005</td>\n",
       "      <td>2.67</td>\n",
       "      <td>shower onli faucet</td>\n",
       "      <td>updat your bathroom with the delta vero single...</td>\n",
       "      <td>0.382979</td>\n",
       "      <td>0.054545</td>\n",
       "      <td>delta vero 1-handl shower onli faucet trim kit...</td>\n",
       "      <td>0.397320</td>\n",
       "      <td>0.098485</td>\n",
       "      <td>0.726722</td>\n",
       "      <td>0.486046</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id                                      product_title  product_uid  \\\n",
       "0   2                     simpson strong-ti 12-gaug angl       100001   \n",
       "1   3                     simpson strong-ti 12-gaug angl       100001   \n",
       "2   9  behr premium textur deckov 1-gal. #sc-141 tugb...       100002   \n",
       "3  16  delta vero 1-handl shower onli faucet trim kit...       100005   \n",
       "4  17  delta vero 1-handl shower onli faucet trim kit...       100005   \n",
       "\n",
       "   relevance         search_term  \\\n",
       "0       3.00        angl bracket   \n",
       "1       2.50           l bracket   \n",
       "2       3.00           deck over   \n",
       "3       2.33    rain shower head   \n",
       "4       2.67  shower onli faucet   \n",
       "\n",
       "                                 product_description  dist_in_title  \\\n",
       "0  not onli do angl make joint stronger, they als...       0.190476   \n",
       "1  not onli do angl make joint stronger, they als...       0.153846   \n",
       "2  behr premium textur deckov is an innov solid c...       0.175000   \n",
       "3  updat your bathroom with the delta vero single...       0.326087   \n",
       "4  updat your bathroom with the delta vero single...       0.382979   \n",
       "\n",
       "   dist_in_desc                                          all_texts  \\\n",
       "0      0.030418  simpson strong-ti 12-gaug angl . not onli do a...   \n",
       "1      0.022901  simpson strong-ti 12-gaug angl . not onli do a...   \n",
       "2      0.017875  behr premium textur deckov 1-gal. #sc-141 tugb...   \n",
       "3      0.048632  delta vero 1-handl shower onli faucet trim kit...   \n",
       "4      0.054545  delta vero 1-handl shower onli faucet trim kit...   \n",
       "\n",
       "   tfidf_cos_sim_in_title  tfidf_cos_sim_in_desc  w2v_cos_sim_in_title  \\\n",
       "0                0.274539               0.182836              0.482381   \n",
       "1                0.000000               0.000000              0.322772   \n",
       "2                0.000000               0.053455              0.307271   \n",
       "3                0.133577               0.043712              0.532707   \n",
       "4                0.397320               0.098485              0.726722   \n",
       "\n",
       "   w2v_cos_sim_in_desc  \n",
       "0             0.425223  \n",
       "1             0.110944  \n",
       "2             0.445303  \n",
       "3             0.465209  \n",
       "4             0.486046  "
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_all.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "搞完之后，我们把不能被『机器学习模型』处理的column给drop掉"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df_all = df_all.drop(['search_term','product_title','product_description','all_texts'],axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: 重塑训练/测试集\n",
    "\n",
    "舒淇说得好，要把之前脱下的衣服再一件件穿回来\n",
    "\n",
    "数据处理也是如此，搞完一圈预处理之后，我们让数据重回原本的样貌"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 分开训练和测试集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df_train = df_all.loc[df_train.index]\n",
    "df_test = df_all.loc[df_test.index]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 记录下测试集的id\n",
    "\n",
    "留着上传的时候 能对的上号"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "test_ids = df_test['id']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 分离出y_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_train = df_train['relevance'].values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 把原集中的label给删去\n",
    "\n",
    "否则就是cheating了"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train = df_train.drop(['id','relevance'],axis=1).values\n",
    "X_test = df_test.drop(['id','relevance'],axis=1).values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5: 建立模型\n",
    "\n",
    "我们用个最简单的模型："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.model_selection import cross_val_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "用CV结果保证公正客观性；并调试不同的alpha值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "params = [1,3,5,6,7,8,9,10]\n",
    "test_scores = []\n",
    "for param in params:\n",
    "    clf = RandomForestRegressor(n_estimators=30, max_depth=param)\n",
    "    test_score = np.sqrt(-cross_val_score(clf, X_train, y_train, cv=5, scoring='neg_mean_squared_error'))\n",
    "    test_scores.append(np.mean(test_score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "画个图来看看："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAhsAAAFyCAYAAABcNBiyAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzt3XecFdX9//HXZ6kCigUFsXzthYAgKBZUxAJqjFiiZNVI\nCzaM/sBeELGhIaCxBaJSjIgtWKPB3gV0VzCAoqLYkCIqKkXKfn5/nNlwuW67u3t3bnk/H495rHfu\n3JnPAO6+95wz55i7IyIiIpIuBXEXICIiIrlNYUNERETSSmFDRERE0kphQ0RERNJKYUNERETSSmFD\nRERE0kphQ0RERNJKYUNERETSSmFDRERE0kphQ0RERNJKYUOklplZbzMrSdhWmtlcM7vdzLaKu75s\nZGYnmNkzZrbEzH4xs6/N7CEz6xa9/7foz3qnCs5xQ3RM2wqOGZf0d5e4rUjHvYnkg/pxFyCSoxwY\nAswHGgMHAecAR5tZW3dfFWNtWcXMxgG9gWJgJLAQ2Bo4AXjBzLoAE4E/A6cC15dzqj8AM919ViWX\nXAX0Byxp/7pq3YCIKGyIpNF/3L04+u+xZvYdMAjoCTxUkxObWSNgtef4SopmdhEhaIxy94uS3h5u\nZqcBa939XTP7BCikjLBhZgcAOwKXVOGya919UjVqbeLuZbZ+VPRebZxfJNOpG0Wk7rxE+G15RwAz\n28zM/mpm75vZT2a2LOoq2CvxQ2bWNWrG72Vm15vZV8ByYONqnOMUM7s26ob4ycweMbNNzKyRmd1q\nZoui/feaWYOKbibqFvrJzBqX8d4kM1tgZha93sfMpkTdICvM7FMzu7eS8zcGLgPmABeXdYy7T3T3\nd6OXE4E9zKxDGYeeCpQAD1Z0zapK6Co7xMzuMrNFwJfRe9dE7+1pZg9EIfP1hM8eZmavm9nPZva9\nmT1uZnsknb/Cc4hkG7VsiNSdXaKvS6OvOwHHAY8AnwEtgbOAV8ysjbsvTPr8EOAXYATQCFgN/CbF\nc1xOCCo3RvX8GVhD+F7QBBgK7A/0jc5XXpcEhNaZc4HfAv8q3WlmGwHHAmPd3c1sS2AKsBgYDvwA\n7ACcWMG5IXQ9bU5o1ahKC87EqP5TgRkJ9RQAJwOvuftXVTgPZrZFGbtXu/tPSfvuItzXMKBptK+0\n1keAjwh/5qWh6wjgGWBeVOtGwPnAG2bW0d2/qOwcIlnJ3bVp01aLG6HZfx3QDdgC2AboBSwBfga2\njo5rUMZntwdWAlcm7OtK+K38Y6Bh0vGpnmMmUC9h/8Ro/9NJ53gT+LQK9/ol8HDSvpOj+z8wet0z\ner13in+Of44+d1wKn5kGfJ60r0d0j/2r8Plx0bFlbc8k/R2XAK8AlnSOodF7/yzj/O8B3wDNE/a1\nA9YC46pyDm3asnFTy4ZIehjwYsJrJwwWLXT3bwDcfc3/Dg6/fW8KrADmAh3LOOd4d1+duKMa55jg\n7okDHacRBk6OTTpuGvBnMytw95Lyb5NHgDOTxhP0Ar5297ei1z8Q/jyOM7P/uvvaCs6XaJPoa3Jr\nQkXuB241s0Pc/bVo36mEFqFHq3iOlYSWmeSWhG+TXjtwt7uX1eriwJjEHWbWCmgP3OTuy/53oPt/\nzex54JjKziGSrTRmQyQ9nPD0yRHAoUAbd9/Z3V8oPcCCQWb2EeGH4beEJvl2QPMyzjk/eUc1zvFl\n0utlFewvKOcciR4idL8cF9XTFDgaeLj0AHd/lfCD/mrg22iMQh8za1jJuX+Mvm5cyXGJHiS0CJwa\n1dMIOJ7QKrGsog8mWOfuL7v7S0nb+2UcO7+C83yW9Pr/oq8flXHsB0CLqAuqonOIZCWFDZH0eSf6\nIfWau88t4/0rCY9yvgKcBnQnhJM5lP3/5spaOEd5j2+Wt7/CcQLuPo3wA/eUaNdxhEd9H0o67hTg\nAOB2oDWhJeVdM2tSwek/jK7frqIakq6zBHgeOMnM6kX1NCN0F6VDWX8nVXmvNs4vkjXUjSISn5OA\nl9z9zMSdZrYpYXxHXZ2jph4GzjezZoQulPnu/k7yQe4+HZgODDGzQkIAKKsLp9QbwPdAoZndWE53\nRVkmEsZpHEN4FPZH4OkU7iddPo++7l7Ge3sA37q7woXkJLVsiMRnHUktB2Z2MmFAaV2eo6YeIjwd\n04fwQ36DVo0o+CSbGX1tVN5Jox+8NwNtgL+UdYyZnWZm+yTtfpzQInAuoUvnX8ljXeLg4cmgGUBv\nMysdj4KFGU27A/+OqzaRdFPLhkh6VOUxxacJv+WPBd4idBecRngssqpq4xw1eqTS3d8zs3nADUBD\nEsZrRHqb2bnAY1FdGwMDCONCnqnk9CMIYWOwhanJHyXMINqKMBZjX+DApHqWm9njhHEbDjyQ4i3V\ntzBZWFkmJ7Q+VOfP7WLCPU+N5hlpApxHaMEZVo3ziWQFhQ2R9KhKk/+NhB82pxLGPBQRmv5vKuPz\n5Z2vNs5RG7OQPgRcAXzs7jOS3nuVEAp6EeYBWUZ42uVUd/+cCkRdJ33M7AngTOBCwlMq3xIez70k\nGjeSbCKhC2WBu7+U4r00Au4r573XgeS5MKrM3V80s6MIwWIYYY6TV4DLKvuzEMlmVvVuUBEREZHU\nVWvMhpkNNLPPLKxmOdXM9q3g2NJpkhO3dZaw+qWZtTGzR6NzlpjZ+WWcZ2gZ55lTnfpFRESk7qQc\nNsysF+FRu6HA3oSBXlPMrEUFH3NgV0I/ayvCDIqLE95vQujLvZQwu155ZhGaYUvPc1Cq9YuIiEjd\nqs6YjUHAGHe/D8DMziasjdCPckaMR5a4+49lveFhIaV3o/PdXME51kbP0YuIiEiWSKllI1oFshMJ\n0zBHA7heIEzYU+5HgRnRKpDPmdmBFRxbkV0trFY5z8zuN7PtqnkeERERqSOptmy0AOoBi5L2L6Ls\niWogdIucRWi5aER45O0VM+tcxqj1ikwlPMc/F9gauAZ4zczauvvy5IOjVRt7EGY3XJXCdURERPJd\nY8LqzFPcfWklx1Yq7Y++uvtHbLgWwFQz25nQHdM7hfNMSXg5y8ymE2bkO4WwUmOyHqRvimIREZF8\ncBqpz1XzK6mGjW8JMxa2TNrfkjDRTlVNB7qkeO0NuPuyaPGpXco5ZD7A/fffz5577lmTS2WMQYMG\nccstt8RdRq3JpfvJpXsB3U8my6V7Ad1Ppvrggw84/fTToeLFBqsspbDh7mvMrAg4HHgSwqqT0evb\nUjhVByp+6qRS0ToMu1D+5DurAPbcc086dixrpe3s07x585y5F8it+8mlewHdTybLpXsB3U8WqJVh\nCNXpRhkFjI9Cx3RCd0gTYDyAmQ0HWrt77+j1BYRlkmcT+oAGAN2AI0tPGA08bUMYSNoQ2MbM2gM/\nu/u86JgRwFOErpNtWD/73qRq3IOIiIjUkZTDhrs/HM2pcS2h+2QG0CPhkdRWQOJTIg0J83K0BlYA\n7wOHu/trCce0Bt5j/fS/F0Xbq8Bh0b5tCf1GWxBWs3wD2L82Bq6IiIhI+lRrgKi73wXcVc57fZNe\njyAsplTR+T6nksdw3b0wxTJFREQkA2iJ+SxSWJhbeSuX7ieX7gV0P5ksl+4FdD/5ImcXYjOzjkBR\nUVFRrg3WERERSavi4mI6deoE0Mndi2t6PrVsiIiISFopbIiIiEhaKWyIiIhIWilsiIiISFrlfNj4\n+OO4KxAREclvOR82rr4a1qyJuwoREZH8lfNh45NP4IYb4q5CREQkf+V82OjfP4SN4ho/JSwiIiLV\nkRdho21b6N0bfvkl7mpERETyT86HjQYNYMIEmDsXhg2LuxoREZH8k/NhA2CvvWDoULj5Zpg2Le5q\nRERE8ktehA2ASy+Fjh1Dd8rKlXFXIyIikj/yJmzUrx+6U+bPhyFD4q5GREQkf+RN2ABo0wauuw5G\njYI33oi7GhERkfyQV2EDYPBg2H9/6NMHli+PuxoREZHcl3dho149GD8eFiyAyy+PuxoREZHcl3dh\nA2C33WD4cLj9dnj55birERERyW15GTYA/vxnOOQQ6NcPfvop7mpERERyV96GjYICGDcOliyBiy+O\nuxoREZHclbdhA2CnnWDECBgzBp57Lu5qREREclNehw2As8+GI44Ia6j88EPc1YiIiOSevA8bZnDv\nvbBsWXgsVkRERGpX3ocNgO23h1tuCWM4nn467mpERERyi8JGpF8/OPpoGDAAvvsu7mpERERyh8JG\nxAzuvhtWrYLzz4+7GhERkdyhsJFgm23gtttg4kR47LG4qxEREckNChtJTj8devaEs84Kc3CIiIhI\nzShsJDGD0aNh3ToYODDuakRERLKfwkYZWrWCu+6CRx6Bhx6KuxoREZHsprBRjl694OST4dxzYeHC\nuKsRERHJXgobFbjrLqhfP8wy6h53NSIiItlJYaMCLVqE8RtPPAH33x93NSIiItlJYaMSJ5wAp50W\nlqT/+uu4qxEREck+ChtVcNtt0KRJmF1U3SkiIiKpUdiogs03D7OLPvssjB0bdzUiIiLZRWGjin77\nW+jbFwYNgs8/j7saERGR7KGwkYJbboHmzaF/f3WniIiIVJXCRgqaN4d774UXXwxPqYiIiEjlFDZS\n1L17WDfl4ovh00/jrkZERCTzVStsmNlAM/vMzFaa2VQz27eCY7uaWUnSts7Mtko4po2ZPRqds8TM\nylzkPZXrptOIEbDllmEMR0lJHBWIiIhkj5TDhpn1AkYCQ4G9gZnAFDNrUcHHHNgVaBVtW7v74oT3\nmwDzgEuBb2rxummx8cbhqZTXXoPbb6/rq4uIiGSX6rRsDALGuPt97v4hcDawAuhXyeeWuPvi0i3x\nDXd/190vdfeHgdW1fN206NYtTPR1+eXw0UdxVCAiIpIdUgobZtYA6AS8WLrP3R14ATigoo8CM8xs\ngZk9Z2YH1tF102r4cNhmG+jTJyxJLyIiIr+WastGC6AesChp/yJC90hZvgHOAk4CTgS+BF4xsw5p\nvm7aNW0K48fD1KkwalRcVYiIiGS2+um+gLt/BCR2NEw1s50J3SK90339QYMG0bx58w32FRYWUlhY\nWCvn79IFBg+GIUPCxF9t2tTKaUVEROrEpEmTmDRp0gb7li1bVqvXSDVsfAusA1om7W8JLEzhPNOB\nLnVx3VtuuYWOHTumcKnUXXcd/Pvf0Ls3vP12WJZeREQkG5T1C3hxcTGdOnWqtWuk1I3i7muAIuDw\n0n1mZtHrt1I4VQfKeeokzddNi402Ct0pxcVw881xVyMiIpJZqvM7+ChgvJkVEVooBhEeXR0PYGbD\ngdbu3jt6fQHwGTAbaAwMALoBR5aeMBoA2oYwkLQhsI2ZtQd+dvd5Vblu3PbbDy69FIYNg2OPhfbt\n465IREQkM6QcNtz94Whui2sJ3RgzgB7uviQ6pBWwXcJHGhLmx2hNeFT1feBwd38t4ZjWwHuE+TgA\nLoq2V4HDqnjd2A0dCk89FbpTpk+Hhg3jrkhERCR+5jm6opiZdQSKioqK0j5mI1FxcWjluOKK0Moh\nIiKSbRLGbHRy9+Kank9ro9Syjh3hyivhhhugqCjuakREROKnsJEGV14J7dqF7pRffom7GhERkXgp\nbKRBgwZw331hGvNrrom7GhERkXgpbKRJu3YhaPzlL2GGURERkXylsJFGl1wCnTqFtVNWroy7GhER\nkXgobKRR/fowYQLMnw9XXRV3NSIiIvFQ2EizPfeE66+HW26B11+PuxoREZG6p7BRBwYNggMOgL59\nYfnyuKsRERGpWwobdaBevbB2yoIFcNllcVcjIiJStxQ26siuu8JNN8Edd8BLL8VdjYiISN1R2KhD\n550HXbtCv37w449xVyMiIlI3FDbqUEEBjB0L334LF18cdzUiIiJ1Q2Gjju20E/z1r/CPf8CUKXFX\nIyIikn4KGzE46yw44gjo3x9++CHuakRERNJLYSMGZnDvvfDTT+GxWBERkVymsBGT7bcPE32NHw9P\nPx13NSIiIumjsBGjvn3hmGNgwABYujTuakRERNJDYSNGZnD33bBqFZx/ftzViIiIpIfCRsxat4bb\nb4cHHoDJk+OuRkREpPYpbGSA006D44+Hs8+GJUvirkZERKR2KWxkADMYPRpKSuCcc8A97opERERq\nj8JGhmjZEu66C/71L3joobirERERqT0KGxnklFPCNnAgLFwYdzUiIiK1Q2Ejw9x5J9SvD2eeqe4U\nERHJDQobGaZFCxgzBp56Cv75z7irERERqTmFjQx0/PFw+ulh7o2vvoq7GhERkZpR2MhQt90GTZrA\nn/6k7hQREcluChsZarPN4J57wjL0994bdzUiIiLVp7CRwY45Bvr1g8GD4fPP465GRESkehQ2Mtyo\nUbDppiF0lJTEXY2IiEjqFDYyXPPmoRvlpZfCLKMiIiLZRmEjCxx5ZFg35eKLYd68uKsRERFJjcJG\nlhgxArbaCvr2VXeKiIhkF4WNLNGsGYwbB6+/Hh6LFRERyRYKG1nk0EPDRF+XXw5z58ZdjYiISNUo\nbGSZ4cNh222hTx9Yty7uakRERCqnsJFlmjSB8eNh2jQYOTLuakRERCqnsJGFunSBCy+EIUNg9uy4\nqxEREamYwkaWuu462Gkn6N0b1qyJuxoREZHyKWxkqcaNYcIEeO89uPnmuKsREREpX7XChpkNNLPP\nzGylmU01s30rOLarmZUkbevMbKuk4042sw+ic840s6OT3h9axnnmVKf+XNG5M1x2GVx7LcycGXc1\nIiIiZUs5bJhZL2AkMBTYG5gJTDGzFhV8zIFdgVbRtrW7L04454HAA8DdQAfgCeBxM2uTdJ5ZQMuE\n8xyUav255uqrYY894IwzYPXquKsRERH5teq0bAwCxrj7fe7+IXA2sALoV8nnlrj74tIt6b3zgWfd\nfZS7z3X3q4Fi4Lyk49a6e+J5vqtG/TmlUaPQnTJnDlx/fdzViIiI/FpKYcPMGgCdgBdL97m7Ay8A\nB1T0UWCGmS0ws+eiloxEB0TnSDSljHPuamZfm9k8M7vfzLZLpf5ctffecNVVcOON8O67cVcjIiKy\noVRbNloA9YBFSfsXEbo1yvINcBZwEnAi8CXwipl1SDimVRXOORXoA/QgtKbsCLxmZk1TvIecdMUV\nsNde4emUVavirkZERGS9tD+N4u4fufvd7v6eu0919/7AW4TumFTOM8Xd/+Xus9z9eeAYYDPglDSU\nnXUaNAjdKR9/DNdcE3c1IiIi69VP8fhvgXWEQZqJWgILUzjPdKBLwuuFqZ7T3ZeZ2UfALhVdaNCg\nQTRv3nyDfYWFhRQWFqZQbnZo1w6GDQtdKscfD/vvH3dFIiKS6SZNmsSkSZM22Lds2bJavYaFIRcp\nfMBsKjDN3S+IXhvwBXCbu4+o4jmeA350999Hrx8ENnL3ngnHvAnMdPdzyzlHs+i6V7v7HWW83xEo\nKioqomPHjindYzZbuzbMMPrDD2EOjiZN4q5IRESyTXFxMZ06dQLo5O7FNT1fdbpRRgEDzOwMM9sD\nGA00AcYDmNlwM5tQerCZXWBmx5nZzmb2GzO7FegGJAaEvwFHmdlgM9vdzK4hDES9I+E8I8zsEDP7\nv2iA6WPAGmDDOJbn6tcPa6d8/nlo4RAREYlbqt0ouPvD0Zwa1xK6OmYAPdx9SXRIKyDxKZGGhHk5\nWhMekX0fONzdX0s459tmdipwQ7R9DPR098RJu7YlzMWxBbAEeAPY392XpnoPuW7PPeGGG+Dii+GE\nE+Dgg+OuSERE8lnK3SjZIl+7UUqtWwddu8I334TZRZs1i7siERHJFpnQjSJZoF49GDcuhI3LLou7\nGhERyWcKGzls113DIm133gkvvlj58SIiIumgsJHjBg6EQw+Ffv3gxx/jrkZERPKRwkaOKyiAsWPh\nu+/goovirkZERPKRwkYe2HFH+Otf4e674T//ibsaERHJNwobeeLMM6F7d/jTn8KEXyIiInVFYSNP\nmME998BPP8EFF8RdjYiI5BOFjTyy3XZw661w333w5JNxVyMiIvlCYSPP9OkDxx4bulWWau5VERGp\nAwobecYMxoyB1avhvPPirkZERPKBwkYeat0abr8dHnwQHn007mpERCTXKWzkqVNPDYu0nXMOLF4c\ndzUiIpLLFDbylBmMHh3++5xzIEfX4xMRkQygsJHHttoK7roLJk8OXSoiIiLpoLCR504+GXr1Cmuo\nfPNN3NWIiEguUtgQ7rwTGjYMj8OqO0VERGqbwoawxRbhcdinnw4TfomIiNQmhQ0BoGdP+OMfw1Tm\nX30VdzUiIpJLFDbkf/72N2jaFPr3V3eKiIjUHoUN+Z/NNguLtT33XPgqIiJSGxQ2ZANHHx1aNgYP\nhvnz465GRERygcKG/MqoUaGVo18/KCmJuxoREcl2ChvyK5tsAmPHwssvw9//Hnc1IiKS7RQ2pExH\nHBGmMb/kEvjkk7irERGRbKawIeX6y1+gZUvo2xfWrYu7GhERyVYKG1KuZs1g3Dh44w247ba4qxER\nkWylsCEV6to1TPR1xRUwd27c1YiISDZS2JBK3XgjbLcd9O4Na9fGXY2IiGQbhQ2pVJMmMH48vPMO\njBwZdzUiIpJtFDakSg48EC68EK6+GmbNirsaERHJJgobUmXXXgs77xy6U9asibsaERHJFgobUmWN\nG8OECTBzJtx0U9zViIhItlDYkJTsuy9cdllo5ZgxI+5qREQkGyhsSMquvhratAndKatXx12NiIhk\nOoUNSVnDhqE7Zc4cuO66uKsREZFMp7Ah1dKhAwwZAsOHh0diRUREyqOwIdV2+eXQvn3oTlm1Ku5q\nREQkUylsSLU1aBC6U+bNC+M4REREyqKwITXSti0MGwZ//Su89Vbc1YiISCZS2JAau+gi6NwZ+vSB\nFSvirkZERDKNwobUWP36Ye2UL78Mq8OKiIgkqlbYMLOBZvaZma00s6lmtm8Fx3Y1s5KkbZ2ZbZV0\n3Mlm9kF0zplmdnRNrit1a4894IYb4G9/g1dfjbsaERHJJCmHDTPrBYwEhgJ7AzOBKWbWooKPObAr\n0Cratnb3xQnnPBB4ALgb6AA8ATxuZm1qeF2pQxdcAAcdBH37ws8/x12NiIhkiuq0bAwCxrj7fe7+\nIXA2sALoV8nnlrj74tIt6b3zgWfdfZS7z3X3q4Fi4LxauK7UkXr1YNw4WLQILrkk7mpERCRTpBQ2\nzKwB0Al4sXSfuzvwAnBARR8FZpjZAjN7LmrJSHRAdI5EU0rPWYPrSh3bZRe4+Wb4+9/hheS/URER\nyUuptmy0AOoBi5L2LyJ0j5TlG+As4CTgROBL4BUz65BwTKtKzlmd60pMzj0XunWD/v3hxx/jrkZE\nROJWP90XcPePgI8Sdk01s50J3SK90339QYMG0bx58w32FRYWUlhYmO5L562CAhg7Ftq1g8GD4Z57\n4q5IRETKM2nSJCZNmrTBvmXLltXqNVING98C64CWSftbAgtTOM90oEvC64WVnLPa173lllvo2LFj\nCqVJbdhhBxg5Es46C046CY7+1bNFIiKSCcr6Bby4uJhOnTrV2jVS6kZx9zVAEXB46T4zs+h1KvNH\ndiB0r5R6O/GckSOj/bV5XalDAwZAjx7wpz/B99/HXY2IiMSlOk+jjAIGmNkZZrYHMBpoAowHMLPh\nZjah9GAzu8DMjjOznc3sN2Z2K9ANuCPhnH8DjjKzwWa2u5ldQxgQmnhMhdeVzGMWulCWLw+PxYqI\nSH5KecyGuz8czW1xLaEbYwbQw92XRIe0ArZL+EhDwvwYrQmPqr4PHO7uryWc820zOxW4Ido+Bnq6\n+5wUrisZaNtt4dZbw9wbJ50EPXvGXZGIiNQ1C0+Q5h4z6wgUFRUVacxGzNzhuOPgnXdg9mzYYou4\nKxIRkYokjNno5O7FNT2f1kaRtDODf/wDVq+GgQPjrkZEROqawobUia23hjvugIcegkceibsaERGp\nSwobUmcKC+HEE8OkX4uTJ6wXEZGcpbAhdcYsTGMOcPbZYSyHiIjkPoUNqVNbbRUCx2OPhUm/FDhE\nRHKfwobUud//PqwKe/HFoYVj9eq4KxIRkXRS2JBY3Hwz3HtvWJK+e3f49tu4KxIRkXRR2JDY9OsH\nL70Ec+ZA584wa1bcFYmISDoobEisDjoIpk+HZs3ggAPgqafirkhERGqbwobEbocd4K234PDDw3Tm\nN9+sgaMiIrlEYUMyQrNmMHkyXHEFXHYZnHEGrFoVd1UiIlIbFDYkYxQUwPXXwwMPwKOPwqGHwjff\nxF2ViIjUlMKGZJzCQnjtNfjiC9h3XygqirsiERGpCYUNyUj77gvvvgutW8PBB8PDD8ddkYiIVJfC\nhmSs1q3h1Vfh+OOhVy8YOhRKSuKuSkREUlU/7gJEKrLRRjBxIrRtC1deCbNnw4QJ0LRp3JWJiEhV\nqWVDMp5ZeErlscfgP/8Jc3N88UXcVYmISFUpbEjWOP74MB/H99+HMR1vvRV3RSIiUhUKG5JV9tor\nzDi6227QrVvoUhERkcymsCFZZ6ut4MUX4Y9/hD59wuqx69bFXZWIiJRHA0QlKzVsCHffDe3aweDB\nYTG3SZNgk03irkxERJKpZUOylhlccAE88wy8+Sbsvz988kncVYmISDKFDcl6PXrAtGmwdi3stx+8\n/HLcFYmISCKFDckJu+8eAkfHjtC9O/z973FXJCIipRQ2JGdsthk8+yyccw6cey4MHAhr1sRdlYiI\naICo5JT69eG228KMowMHwocfwiOPwOabx12ZiEj+UsuG5KQzz4QXXoCZM6FzZ/jgg7grEhHJXwob\nkrO6dg0TgDVuHJ5UeeaZuCsSEclPChuS03baKUxrfsghcOyxMHIkuMddlYhIflHYkJy3ySbw+ONw\nySVw0UXQrx/88kvcVYmI5A+FDckL9erBTTfBP/8ZZho97DBYtCjuqkRE8oPChuSV00+HV16BefPC\nyrEzZsRdkYhI7lPYkLyz//7wzjuw5ZbQpQtMnhx3RSIiuU1hQ/LSdtvB66+HQaMnnQTXXaeBoyIi\n6aJJvSRvNWkCDz4YJgC7+mqYNQvGjQv7RUSk9qhlQ/KaGQwZAo8+Ck8/HR6R/eqruKsSEcktChsi\nhK6UN9/9VjnUAAAasklEQVSExYvDwNFp0+KuSEQkdyhsiEQ6dAgDR3fcMcw+ev/9cVckIpIbFDZE\nErRsCS+/DH/4A/zxj3D55VBSEndVIiLZTQNERZI0ahQGirZrBxdfDLNnw8SJsPHGcVcmIpKd1LIh\nUgYzuPDCMGj0lVfgwAPhs8/irkpEJDtVK2yY2UAz+8zMVprZVDPbt4qf62Jma8ysOGl/fTO72sw+\nic75npn1SDpmqJmVJG1zqlO/SFUdcwxMnQorV4aBo6++GndFIiLZJ+WwYWa9gJHAUGBvYCYwxcxa\nVPK55sAE4IUy3r4BGAAMBPYExgCPmVn7pONmAS2BVtF2UKr1i6SqTZvwdMpee8ERR8Ddd8ddkYhI\ndqlOy8YgYIy73+fuHwJnAyuAfpV8bjQwEZhaxnunAze4+xR3n+/uo4FngAuTjlvr7kvcfXG0fVeN\n+kVStsUWMGUKDBgAZ54JF1wAa9fGXZWISHZIKWyYWQOgE/Bi6T53d0JrxQEVfK4vsCMwrJxDGgHJ\ni36v5NctF7ua2ddmNs/M7jez7VKpX6QmGjSAu+6CO+8M2zHHwPffx12ViEjmS7VlowVQD0henHsR\noVvjV8xsV+BG4DR3L+8hwinAYDPbxYIjgROBrROOmQr0AXoQWlN2BF4zs6Yp3oNIjZx7Ljz3HLz7\nLuy3H8ydG3dFIiKZLa2PvppZAaHrZKi7zyvdXcahFwD/AD4ESoB5wFgSumbcfUrC8bPMbDrwOXAK\nMK68GgYNGkTz5s032FdYWEhhYWHK9yNS6rDDYPp0OO64EDgefhi6d4+7KhGR1E2aNIlJkyZtsG/Z\nsmW1eg3zFJa6jLpRVgAnufuTCfvHA83d/YSk45sD3wNrWR8yCqL/Xgt0d/dXEo5vCGzh7t+Y2U3A\nb929XQX1TAeed/cry3ivI1BUVFREx44dq3yPIqlYtgwKC8N4jlGj4Pzzw2OzIiLZrLi4mE6dOgF0\ncvfiyo6vTErdKO6+BigCDi/dZ2YWvX6rjI/8CLQFOgDto200oQWjPbDBChTuvjoKGg2Ak4DHy6vF\nzJoBuwDfpHIPIrWpeXN46ikYPBj+3/8Lg0dXr467KhGRzFKdbpRRwHgzKwKmE55OaQKMBzCz4UBr\nd+8dDR7dYC4MM1sMrHL3DxL2dQa2AWYA2xIeqzVgRMIxI4CnCF0n2xAGm64BNmz7Ealj9erBiBHw\nm9/AWWeFMRz/+hdsuWXclYmIZIaUH31194eBi4BrgfeAvYAe7r4kOqQVkOpTIo2B64HZwL+AL4GD\n3P3HhGO2BR4gtIo8CCwB9nf3paneg0g69OkT1lWZOzdMAPb++3FXJCKSGVIas5FNNGZD4vLFF2Hg\n6CefhDVVevaMuyIRkdTEOmZDRCq3/fbw5pvQoweccAIMHw45mulFRKpEYUMkDZo2hUcegSFD4Ior\n4PTTw/oqIiL5SGFDJE0KCmDYMHjoIXjsMejaFRYsiLsqEZG6p7AhkmannAKvvx6Cxr77hplHRUTy\nicKGSB3o1AneeQe23RYOPhgefDDuikRE6o7Chkgd2XprePVV+P3vw6yjQ4ZASXmrBYmI5JC0ro0i\nIhtq3Bjuuw/atoXLL4fZs8PrZs3irkxEJH3UsiFSx8zg0kvhiSfg+eehSxf4/PO4qxIRSR+FDZGY\n/O538Pbb8OOPYeDom2/GXZGISHoobIjEqG3bMHB0zz2hWzcYOzbuikREap/ChkjMWrQI3Sl9+kD/\n/nDhhbBuXdxViYjUHg0QFckADRvCmDHQrh0MGgRz5oTHY5s3j7syEZGaU8uGSIYwgz//GZ59FqZO\nhf33h48/jrsqEZGaU9gQyTBHHgnTpoU5OPbbD154Ie6KRERqRmFDJAPttlsIHJ07w1FHwZ13auVY\nEcleChsiGWrTTeHpp0PXynnnwTnnwJo1cVclIpI6hQ2RDFa/PtxyC9xzT3gstnt3WLo07qpERFKj\nsCGSBfr3hxdfhFmzQtfK7NlxVyQiUnUKGyJZ4uCDwwRgTZvCAQeELhYRkWygsCGSRXbYIUxrfthh\ncNxxMGKEBo6KSOZT2BDJMhtvDJMnh1VjL7kEeveGVavirkpEpHwKGyJZqKAAbrgBJk6Ehx8O66os\nXBh3VSIiZVPYEMlip54Kr70Wlqjfd1947724KxIR+TWFDZEs17lzGDjaqhV06QKPPBJ3RSIiG1LY\nEMkB22wTWjh69oRTToFhw8J05yIimUCrvorkiI02ggcegLZt4aqrwpwc48eHR2VFROKklg2RHGIG\nV14ZnlZ59tkwN8eXX8ZdlYjkO4UNkRx0wglhPo6lS8PA0bffjrsiEclnChsiOap9+zBwdNdd4dBD\n4b774q5IRPKVwoZIDttqK3jhBTjttDD51yWXwLp1cVclIvlGA0RFclyjRnDvvdCuHVx0EcyZEwaS\nbrJJ3JWJSL5Qy4ZIHjCDQYPg3/+G118PC7l9+mncVYlIvlDYEMkjRx0F06bB6tVh4OjLL8ddkYjk\nA4UNkTyzxx4hcOy9N3TvDmPGxF2RiOQ6hQ2RPLT55mEejrPPDtt558GaNXFXJSK5SgNERfJUgwZw\n++1hxtHzzoMPPwwryG6+edyViUiuUcuGSJ476yx4/vmwYux++8EHH8RdkYjkGoUNEeHQQ8MEYI0a\nwf77h5VjNR+HiNQWhQ0RAWCnneCtt0LwOOUU2HlnuPFGWLgw7spEJNspbIjI/2yyCTz+OEydCt26\nwXXXwXbbhfDx0kvgHneFIpKNqhU2zGygmX1mZivNbKqZ7VvFz3UxszVmVpy0v76ZXW1mn0TnfM/M\netTWdUWk6szC2I1x42DBAhg5MixXf/jh4bHZkSPDAm8iIlWVctgws17ASGAosDcwE5hiZi0q+Vxz\nYALwQhlv3wAMAAYCewJjgMfMrH1Nrysi1bfZZnD++TB7Nrz6KnTqBJdfDttsA3/8Y1hZVq0dIlKZ\n6rRsDALGuPt97v4hcDawAuhXyedGAxOBqWW8dzpwg7tPcff57j4aeAa4sBauKyI1ZAaHHBLWVPnq\nK7j22jC+46CDYK+94M47YdmyuKsUkUyVUtgwswZAJ+DF0n3u7oTWigMq+FxfYEdgWDmHNAJ+Sdq3\nEjioJtcVkdq31VZh9diPP4YpU8IS9hdcAK1bw4ABUFQUd4UikmlSbdloAdQDFiXtXwS0KusDZrYr\ncCNwmruXlHPeKcBgM9vFgiOBE4Gtq3tdEUmvgoIw3fnkyfDFF3DZZfCf/8A++4R1V+65B5Yvj7tK\nEckEaX0axcwKCF0nQ919XunuMg69APgY+JDQwnEbMBYoL5yISAZp3RqGDIHPPoMnnwytH2eeGfaf\ndx78979xVygicUp1uvJvgXVAy6T9LYGynsbfGNgH6GBmd0b7CgAzs9VAd3d/xd2/BU40s4bAFu7+\njZndBJQugp3qdf9n0KBBNG/efIN9hYWFFBYWVvQxEamG+vXhd78L2/z5cPfdcO+9YUxHly5hHZbf\n/x4aN467UhEpNWnSJCZNmgSEAd8//ADz59fuICzzFIeSm9lUYJq7XxC9NuAL4DZ3H5F0rBGeLkk0\nEOgGnATMd/eVZVyjATAHeNDdh6R63ej9jkBRUVERHTt2TOkeRaT2rF4NTzwBo0eHuTo23xz69g0t\nH7vtFnd1IrJ2Lbz2WugSfeyx8Mj7ppsW88MPnQA6uXtxZeeoTHW6UUYBA8zsDDPbg/CUSRNgPICZ\nDTezCRAGcbr7nMQNWAyscvcPSoOGmXU2sxPMbEczOxh4ltDdMqKq1xWRzNSwIZx8Mrz4IsydG4LG\nuHGw++5wxBFhavTVq+OuUiS/rFoF//439OsHrVqFeXSefDL8v/rqq/Dcc7V7vZTDhrs/DFwEXAu8\nB+wF9HD3JdEhrYDtUjxtY+B6YDbwL+BL4CB3/zGF64pIhtttN/jrX+Hrr+Gf/wzf8E45BbbfHq68\nMnS9iEh6/PxzCPeFhWFc1bHHhrlyzjwzrI30+edw663hMfd69Wr32il3o2QLdaOIZIf//hfGjAnh\n46ef4Oijw9iOY46p/W94Ivlm6VJ46qnQPTJlCvzyC+y9N5x4Ytj23DPMo5OsuLiYTp3i7UYREak1\n7drBHXeEfuK774ZFi+C442DHHcPaLAsWxF2hpIN7aMn6+GMo0XOHtWrBArjrrtBN2bJl6CpZujQs\nrPjpp1BcDFddBW3alB000iHVp1FERNKiaVPo3z9s774bWjtuugmGDYOePeGss8I3zwL9ipR1li8P\n6+vMnAnvv7/+649RR/mmm4a5WfbbDzp3DlvL5GcPpUKffhoGeE6eDG+/HVoFDzssBPmePWHrrSs/\nRzqpG0VEMtayZXD//eFJllmzYKedQujo2xe23DLu6iSZe+j3Tw4Vn3wS3qtXLwwM3msvaN8+fK1f\nH6ZPD9u0abB4cTjX//3f+uCx337QsWMIpBK4w5w56wPGjBnhkfIePUL3yLHHhie/qqu2u1EUNkQk\n47mHtVhGjw4D3NzhpJPC2I6DD667pmBZb/nyMN4mMVQktlZsvnkIFKWhon370Gxf0Rwr7mE22sTw\nUVQEK1aEoNK27YYBpE2b/BrX4x5a/UoDxkcfwcYbh2Bx4olw1FHQrFntXEtho4oUNkRy09KlMGFC\nCB4ffxwGuJ19dliFdrPN4q4u95SOrUgMFTNnwrx55bdWtG8fZo+tjRC4dm34DX7atPUBZPbsMM6j\nadOwEnFi98t22+VW+Fy3Dt54Y33A+Oor2GKL0DVy0knhkdVGjWr/ugobVaSwIZLb3OHll0PoeOwx\naNAA/vCH0M3SuXNu/cCpKz///OuxFf/9b81aK9JVZ3Hx+gAyfXpoEYEwZ0Ri68c++4QxIdnkl1/C\nBHiTJ4cJ8ZYsgW22Ca0XJ5wQWvPqp3nEpcJGFSlsiOSPhQth7Fj4xz/CmIEOHUJrx6mnhmZm2VBp\na0Xy2Irk1orEULHXXrXXWpEO33wT5ooobf14550w5gdgjz02DCB77RUmm8sky5eHhQwnT4annw4B\nb5ddQuvFiSeG0FSXg6MVNqpIYUMk/6xbF+YSGD06zI7YpAmcdlr4Rt2sWQgeyVuzZuG4TP0hWlM/\n/1z22Iqffgrvb7HFr0NFHK0Vta2kJHSzJbZ+zJgBa9aEoLH33ht2v+yyS93/G/j++xAsJk8OQWPV\nqvB3UDoHxm9+E9+/S4WNKlLYEMlvX34ZlrkfOzb0c1ekoKDsMFJRQClrf+l7G21U9z8kSkrKH1sB\nG7ZWJIaLrbfO3aCVbNWq8GdS2voxfXoIJBDG+5S2fJQGkHQ88bRwYegamTw5dJWsXQsHHLC+i2Tn\nnWv/mtWhsFFFChsiUqqkJPyG/9NP67+WtZX3XvL+FSsqvl69er8OJNUNLhtvHFoZEgPBTz+VPbYi\n11sr0uG77zbsfpk+PYyRgDCxXGL3y957h1awVM2fH8YVTZ4cpgcvKICuXUPAOP74MB4j09R22NCk\nXiKS8woKYJNNwlYb1q2rfniZP//X76381drXG6pXb30AKSgI41JK9++xRwgTxx2Xn60VNbX55mFu\nih49wuvSuUISu1+uuir8HdWrF/6MEwPIHnuU/fjtBx+sf4KkuDg8MdK9O9x7L/zud9CiRd3eZ9wU\nNkREUlSvHjRvHrbasHZt1cPLmjXhB1z79uGxX7VW1C4z2GGHsPXqFfatWRMety1t/XjzzTAY2T20\nPu2zz/qWj/ffDwHjww/De7/9LVx6aVjzJ58HKytsiIjErH798Hhmtj2imS8aNAhPOHXoEFZIhRD+\niorWt35MnAg33xxaSnr2hBEjwvT6CoOBwoaIiEiKNt4YDj00bKWWLAmBsUGDuKrKXAobIiIitUDr\n9ZRP6yeKiIhIWilsiIiISFopbIiIiEhaKWyIiIhIWilsiIiISFopbIiIiEhaKWyIiIhIWilsiIiI\nSFopbIiIiEhaKWyIiIhIWilsiIiISFopbIiIiEhaKWyIiIhIWilsiIiISFopbIiIiEhaKWyIiIhI\nWilsiIiISFopbIiIiEhaKWyIiIhIWilsiIiISFopbIiIiEhaKWyIiIhIWilsiIiISFopbIiIiEha\nKWyIiIhIWilsZJFJkybFXUKtyqX7yaV7Ad1PJsulewHdT76oVtgws4Fm9pmZrTSzqWa2bxU/18XM\n1phZcRnvXWhmc81shZl9YWajzKxRwvtDzawkaZtTnfqzVa79I86l+8mlewHdTybLpXsB3U++qJ/q\nB8ysFzASOBOYDgwCppjZbu7+bQWfaw5MAF4AWia9dwZwPdAHeBvYDRgPlAAXJRw6CzgcsOj12lTr\nFxERkbpVnZaNQcAYd7/P3T8EzgZWAP0q+dxoYCIwtYz39gXecPeH3P0Ld38BeBDonHTcWndf4u6L\no+27atQvIiIidSilsGFmDYBOwIul+9zdCa0VB1Twub7AjsCwcg75D7B3aXeMme0EHAP8O+m4Xc3s\nazObZ2b3m9l2qdQvIiIidS/VbpQWQD1gUdL+RcDuZX3AzHYFbgQOcvcSM/vVMe7+bzMbCrxlZh5d\nY7S735xw2FRCN8tcYGvgGuA1M2vr7svLuHRjgA8++KDKN5fpli1bRnHxr4a7ZK1cup9cuhfQ/WSy\nXLoX0P1kqoSfnY1r5YTuXuWN8EO+BNgvaf/NwNtlHF9AGNdxZsK+a4DipONOAH4A+gK/AXoCnwNX\nVVBL89LPlPP+qYBr06ZNmzZt2qq9nZpKTihvS7Vl41tgHUkDPKPXC8s4fmNgH6CDmd0Z7SsAzMxW\nA93d/RXgUuAf7j4uOma2mTUDxhAGjv6Kuy8zs4+AXcqpdQpwGjAfWFX5rYmIiEikMbAD4WdpjaUU\nNtx9jZkVEZ4IeRJCaohe31bGR34E2ibtGwh0A04iBAEIAST5yZKS0vNH40I2EIWRXYD7yql1KfBA\npTclIiIiZXmrtk6U8qOvwChgfBQ6Sh99bUJ4VBUzGw60dvfeUUjYYC4MM1sMrHL3xMEUjwOXmNlM\nYBqwK3At8GRp0DCzEcBThO6VbQiDTdcAeqhZREQkg6UcNtz9YTNrQQgDLYEZQA93XxId0gpI9SmR\nm6Kv1wLbAksILSdXJRyzLaGlYovo/TeA/aMWDBEREclQVkYPhYiIiEit0dooIiIiklYKGyIiIpJW\nORk2zOxgM3symm20xMyOi7um6jKzy81supn9aGaLzOwxM9st7rqqw8zONrOZZrYs2t4ys6Pirqu2\nmNll0b+3UXHXUh25ttihmbU2s3+a2bfRAo8zzaxj3HVVR7TwZfLfTYmZ3R53bdVhZvXMbHh0XyvM\n7BMzu6ryT2YmM2tmZrea2fzoft4ws33irqsqqvLz0syuNbMF0b09b2blTTlRrpwMG0BTwsDVcwmT\nkmSzg4Hbgf2AI4AGwHNmtlGsVVXPl4Q5VToSpr1/CXjSzNrEWlUtiKbaPxOYGXctNTSLMPC7VbQd\nFG851WNmmwJvAr8APYA9gQuB7+Osqwb2Yf3fSSvgSML3tofjLKoGrgT6A+cAewCXEJ5IPC/Wqqrv\nXsIUEKcRpnt4HnjBzLaOtaqqqfDnpZldCpxH+P7WGVhOWHy1YSoXyfkBomZWAhzv7k/GXUttiJ4E\nWgwc4u5vxF1PTZnZUuCihAndsk4050sR4RvnEOA9dx8cb1Wpi5YM6OnuWfnbfyIzuwk4wN27xl1L\nOpjZrcAx7p6trZxPAQvdfUDCvkeBFe5+RnyVpc7MGgM/Ab9z9/8k7H8XeMbdr46tuBSV9fPSzBYA\nI9z9luj1JoQlSnq7e5XDbq62bOSyTQnpM6tXvDWzAjP7A9AIeD3uemroTuApd38p7kJqQa4sdvg7\n4F0zezjqfiw2sz/FXVRtiBbEPI3w23S2ehY4PFo7CzNrD3QBnom1quqpT1jP65ek/SvJ0pbBUma2\nI6ElLXHx1R8J82GVu/hqWaozqZfEJJqt9VbgDXfPyr50M2sLvE2YCncFcIq7fxJvVdUXBaYOhGbu\nbJfqYoeZbCdCS9NI4AZC8+9tZvaLu/8z1spq7gTC2lAT4i6kutz9rijIzjWztYRffK909wdjLi1l\n7v6zmb0NDDGzDwm/9Z9K+GH8cazF1Vwrwi+3ZS2+2iqVEylsZJe7gDaE3wCy1YdAe8I3y98DD5pZ\nV3d/L96yUmdm2xLC3xHuvibuemrK3RPXQJhlZtMJM/aeAmRbN1cBMN3dh0SvZ0ZB92wg28NGP+BZ\ndy9rPaqsYGbnA72BXoRZpjsAfzOzBVkaBk8HxgJfE5beKCZMQtkpzqIyicJGljCzO4BjgIPd/Zu4\n66kud18LfBq9fM/MOhN+Az0zvqqqrROwJVActTpBaE49JBro1qisdX2yRRUWO8xk3wAfJO37ADgx\nhlpqjZltTxgofnzctdTQFcAwd38kej3bzHYALicLw6C7fwZ0iwbub+Lui8zsQdZ/r8tWCwEjDBpP\nbN1oCaT0C6LGbGSBKGj0BLq5+xdx11PLCgg/oLPRC0A7wm9l7aPtXeB+oH02Bw3YYLHDbAy3bwK7\nJ+3bndBSk836Eb7pZ+PYhkQFhBXEE5WQ5T+T3H1lFDQ2IzwF9XjcNdVEFKIWEp60Af43QHQ/Ulyk\nLSdbNsysKeGbZOlvmztFA5C+c/cv46ssdWZ2F1AIHAcsN7OW0VvL3H1VfJWlzsxuJAwM+wLYmDDI\n7RDg+jjrqq5oHEPyQoPLgaVJCw1mBcutxQ5vAd40s8sJj4fuB/wJGFDhpzJY1HrWBxjv7iUxl1NT\njwNXmdlXwGzC4/CDgHtiraqazKw74efNXMJCon8hfG8YH2NZVVKFn5e3Ev6uPiGs1H4d8BXwREoX\ncvec24CuhJS8LmkbG3dt1biXsu5jHXBG3LVV417uITQrriSk5eeAw+Kuq5bv8SVgVNx1VLP2SdE3\nkZWEQPgAsGPcddXgfo4B3icMRJ4N9Iu7phrez5HR//u7xF1LLdxLE2BE9P1gOWEg5TCgfty1VfN+\nTgY+if7f+Rr4G7Bx3HVVsfZKf14SBosviP5fmlKdf4M5P8+GiIiIxCur+8dEREQk8ylsiIiISFop\nbIiIiEhaKWyIiIhIWilsiIiISFopbIiIiEhaKWyIiIhIWilsiIiISFopbIiIiEhaKWyIiIhIWils\niIiISFr9f0Ue62Wh2yzxAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1c3231080>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "plt.plot(params, test_scores)\n",
    "plt.title(\"Param vs CV Error\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "大概9的时候达到了最优解。直观的来看 可能跟前文的差别不大。\n",
    "\n",
    "但是同学们可以试试其他不同的算法，就可以对比出差距了。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 6: 上传结果\n",
    "\n",
    "用我们测试出的最优解建立模型，并跑跑测试集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "rf = RandomForestRegressor(n_estimators=30, max_depth=6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomForestRegressor(bootstrap=True, criterion='mse', max_depth=6,\n",
       "           max_features='auto', max_leaf_nodes=None,\n",
       "           min_impurity_split=1e-07, min_samples_leaf=1,\n",
       "           min_samples_split=2, min_weight_fraction_leaf=0.0,\n",
       "           n_estimators=30, n_jobs=1, oob_score=False, random_state=None,\n",
       "           verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 121,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_pred = rf.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "把拿到的结果，放进PD，做成CSV上传："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pd.DataFrame({\"id\": test_ids, \"relevance\": y_pred}).to_csv('submission.csv',index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 总结：\n",
    "\n",
    "这一篇教程中，用了稍微复杂的方法教大家加入更多高质量的features。\n",
    "\n",
    "同学们可以尝试修改/调试/升级的部分是：\n",
    "\n",
    "2. **更多的特征**: Deep Learning的大重点就是黑盒机制。所以，其实我们刚刚得到的tfidf或者w2v vector可以就不做任何处理塞进我们的X里面去。然后我们使用更加深度的学习模型来训练它。这样，我们的算法可以“看见”更加原始的信息。假设他们比人脑好的话，他们就可以作出更好的predictions。\n",
    "\n",
    "3. **更好的回归模型**: 根据之前的课讲的Ensemble方法，把分类器提升到极致"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
